Skip to main content

Focused Crawls

Internet Archive

Focused crawls are collections of frequently-updated webcrawl data from narrow (as opposed to broad or wide) web crawls, often focused on a single domain or subdomain.



rss RSS

Show sorted alphabetically
Show sorted alphabetically
SHOW DETAILS
up-solid down-solid
eye
Title
Date Archived
Creator
Top Domains
collection
176,109
ITEMS
2.1B
VIEWS
collection
eye 2.1B
A daily collection of thousands of the most popular web sites according to Alexa.com's top sites rankings .
Topics: daily, popular sites, Alexa
Top News
collection
125,073
ITEMS
855.1M
VIEWS
collection
eye 855.1M
A daily collection of hundreds of the world's top news sites.
Topics: daily, news
top_domains-02750
top_domains-02750
collection
1,897
ITEMS
36.7M
VIEWS
by Internet Archive
collection
eye 36.7M
top_domains-00900
top_domains-00900
collection
4,129
ITEMS
62.7M
VIEWS
by Internet Archive
collection
eye 62.7M
taringa.net
taringa.net
collection
1,040
ITEMS
22.8M
VIEWS
by taringa.net
collection
eye 22.8M
top_domains-03000
top_domains-03000
collection
1,837
ITEMS
26.2M
VIEWS
by Internet Archive
collection
eye 26.2M
top_domains-00500
top_domains-00500
collection
3,094
ITEMS
78.8M
VIEWS
by Internet Archive
collection
eye 78.8M
foxnews.com
foxnews.com
collection
1,055
ITEMS
18.3M
VIEWS
by http://foxnews.com
collection
eye 18.3M
top_domains-05250
top_domains-05250
collection
3,155
ITEMS
22.3M
VIEWS
by Internet Archive
collection
eye 22.3M
top_domains-00300
top_domains-00300
collection
4,822
ITEMS
54.4M
VIEWS
by Internet Archive
collection
eye 54.4M
top_domains-02500
top_domains-02500
collection
2,887
ITEMS
37.7M
VIEWS
by Internet Archive
collection
eye 37.7M
top_domains-09250
top_domains-09250
collection
1,245
ITEMS
14.8M
VIEWS
by Internet Archive
collection
eye 14.8M
top_domains-00250
top_domains-00250
collection
2,074
ITEMS
51.8M
VIEWS
by Internet Archive
collection
eye 51.8M
top_domains-01500
top_domains-01500
collection
2,691
ITEMS
44.6M
VIEWS
by Internet Archive
collection
eye 44.6M
top_domains-00450
top_domains-00450
collection
2,899
ITEMS
39M
VIEWS
by Internet Archive
collection
eye 39M
top_domains-03250
top_domains-03250
collection
955
ITEMS
18.3M
VIEWS
by Internet Archive
collection
eye 18.3M
yahoo.com
yahoo.com
collection
1,280
ITEMS
37.7M
VIEWS
collection
eye 37.7M
buzzfeed.com
buzzfeed.com
collection
2,073
ITEMS
19.2M
VIEWS
by buzzfeed.com
collection
eye 19.2M
nytimes.com
nytimes.com
collection
1,149
ITEMS
23.8M
VIEWS
by http://nytimes.com
collection
eye 23.8M
top_domains-00350
top_domains-00350
collection
2,157
ITEMS
49.2M
VIEWS
by Internet Archive
collection
eye 49.2M
nbcnews.com
nbcnews.com
collection
867
ITEMS
25.3M
VIEWS
by nbcnews.com
collection
eye 25.3M
top_domains-00700
top_domains-00700
collection
1,585
ITEMS
41.2M
VIEWS
by Internet Archive
collection
eye 41.2M
top_domains-02000
top_domains-02000
collection
2,229
ITEMS
33.9M
VIEWS
by Internet Archive
collection
eye 33.9M
top_domains-08000
top_domains-08000
collection
1,342
ITEMS
27.6M
VIEWS
by Internet Archive
collection
eye 27.6M
top_domains-01250
top_domains-01250
collection
918
ITEMS
18M
VIEWS
by Internet Archive
collection
eye 18M
top_domains-01000
top_domains-01000
collection
2,708
ITEMS
29.7M
VIEWS
by Internet Archive
collection
eye 29.7M
top_domains-03500
top_domains-03500
collection
1,489
ITEMS
19.6M
VIEWS
by Internet Archive
collection
eye 19.6M
top_domains-04500
top_domains-04500
collection
1,852
ITEMS
27.2M
VIEWS
by Internet Archive
collection
eye 27.2M
top_domains-06500
top_domains-06500
collection
1,943
ITEMS
19.9M
VIEWS
by Internet Archive
collection
eye 19.9M
top_domains-00400
top_domains-00400
collection
2,170
ITEMS
37.4M
VIEWS
by Internet Archive
collection
eye 37.4M
top_domains-03750
top_domains-03750
collection
1,764
ITEMS
20.8M
VIEWS
by Internet Archive
collection
eye 20.8M
top_domains-06000
top_domains-06000
collection
3,062
ITEMS
24.9M
VIEWS
by Internet Archive
collection
eye 24.9M
top_domains-02250
top_domains-02250
collection
2,400
ITEMS
39M
VIEWS
by Internet Archive
collection
eye 39M
abc.net.au-news
abc.net.au-news
collection
1,021
ITEMS
5M
VIEWS
by http://abc.net.au/news/
collection
eye 5M
edgi monitor
edgi monitor
collection
2,156
ITEMS
21.3M
VIEWS
collection
eye 21.3M
welt.de
welt.de
collection
1,557
ITEMS
15.3M
VIEWS
by http://welt.de/
collection
eye 15.3M
top_domains-09000
top_domains-09000
collection
2,777
ITEMS
23.5M
VIEWS
by Internet Archive
collection
eye 23.5M
top_domains-07500
top_domains-07500
collection
2,281
ITEMS
34.6M
VIEWS
by Internet Archive
collection
eye 34.6M
rtl.de
rtl.de
collection
1,431
ITEMS
4.5M
VIEWS
by http://rtl.de/
collection
eye 4.5M
top_domains-04000
top_domains-04000
collection
2,525
ITEMS
26.9M
VIEWS
by Internet Archive
collection
eye 26.9M
top_domains-04750
top_domains-04750
collection
1,402
ITEMS
22M
VIEWS
by Internet Archive
collection
eye 22M
top_domains-00600
top_domains-00600
collection
6,460
ITEMS
58.7M
VIEWS
by Internet Archive
collection
eye 58.7M
worldpress.org
worldpress.org
collection
466
ITEMS
35.8M
VIEWS
by http://worldpress.org/
collection
eye 35.8M
top_domains-01750
top_domains-01750
collection
1,104
ITEMS
23.4M
VIEWS
by Internet Archive
collection
eye 23.4M
top_domains-05500
top_domains-05500
collection
6,671
ITEMS
21.5M
VIEWS
by Internet Archive
collection
eye 21.5M
top_domains-08250
top_domains-08250
collection
3,292
ITEMS
19M
VIEWS
by Internet Archive
collection
eye 19M
themeforest.net
themeforest.net
collection
486
ITEMS
25.7M
VIEWS
by themeforest.net
collection
eye 25.7M
top_domains-00800
top_domains-00800
collection
7,418
ITEMS
48.4M
VIEWS
by Internet Archive
collection
eye 48.4M
top_domains-09500
top_domains-09500
collection
3,700
ITEMS
30.1M
VIEWS
by Internet Archive
collection
eye 30.1M
top_domains-08750
top_domains-08750
collection
1,913
ITEMS
26.1M
VIEWS
by Internet Archive
collection
eye 26.1M
top_domains-06250
top_domains-06250
collection
1,385
ITEMS
22.3M
VIEWS
by Internet Archive
collection
eye 22.3M
top_domains-04250
top_domains-04250
collection
2,119
ITEMS
26.2M
VIEWS
by Internet Archive
collection
eye 26.2M
seattlepi.com
seattlepi.com
collection
1,466
ITEMS
9.3M
VIEWS
by http://www.seattlepi.com/
collection
eye 9.3M
top_domains-06750
top_domains-06750
collection
2,990
ITEMS
22.5M
VIEWS
by Internet Archive
collection
eye 22.5M
bild.de
bild.de
collection
2,548
ITEMS
5.4M
VIEWS
by http://bild.de/
collection
eye 5.4M
top_domains-07750
top_domains-07750
collection
979
ITEMS
14.3M
VIEWS
by Internet Archive
collection
eye 14.3M
abcnews.go.com
abcnews.go.com
collection
1,611
ITEMS
9.8M
VIEWS
by http://abcnews.go.com/
collection
eye 9.8M
time.com
time.com
collection
612
ITEMS
26.4M
VIEWS
by http://time.com
collection
eye 26.4M
wordpress.org
wordpress.org
collection
1,252
ITEMS
14.9M
VIEWS
by http://wordpress.org
collection
eye 14.9M
example.com
example.com
collection
1,333
ITEMS
3.2M
VIEWS
by https://example.com
collection
eye 3.2M
This is not an archive of example.com. This is a "focused crawl" (H3 deriver module) test using a "seedsfile" CXML template with a large (5-10K) seed list to determine how much we can crawl in a day, and to investigate limitations, constraints, or bottlenecks in this context.
sfgate.com
sfgate.com
collection
3,382
ITEMS
18M
VIEWS
by http://sfgate.com
collection
eye 18M
top_domains-05000
top_domains-05000
collection
942
ITEMS
16.1M
VIEWS
by Internet Archive
collection
eye 16.1M
top_domains-09750
top_domains-09750
collection
1,419
ITEMS
14.7M
VIEWS
by Internet Archive
collection
eye 14.7M
top_domains-05750
top_domains-05750
collection
993
ITEMS
16.2M
VIEWS
by Internet Archive
collection
eye 16.2M
top_domains-07250
top_domains-07250
collection
1,974
ITEMS
14.8M
VIEWS
by Internet Archive
collection
eye 14.8M
weather.com
weather.com
collection
628
ITEMS
9.9M
VIEWS
by http://weather.com/
collection
eye 9.9M
seattlepi.com
web
eye 112,281
favorite 0
comment 0
Source: seattlepi.com
top_domains-07000
top_domains-07000
collection
1,932
ITEMS
21.7M
VIEWS
by Internet Archive
collection
eye 21.7M
washingtonpost.com
washingtonpost.com
collection
1,126
ITEMS
13.3M
VIEWS
by http://washingtonpost.com/
collection
eye 13.3M
stumbleupon.com
stumbleupon.com
collection
170
ITEMS
23.2M
VIEWS
by http://stumbleupon.com
collection
eye 23.2M
blogspot.in
blogspot.in
collection
206
ITEMS
13.6M
VIEWS
by blogspot.in
collection
eye 13.6M
pinterest.com
pinterest.com
collection
247
ITEMS
15.9M
VIEWS
by pinterest.com
collection
eye 15.9M
salon.com
salon.com
collection
1,858
ITEMS
14.5M
VIEWS
by http://salon.com/
collection
eye 14.5M
wix.com
wix.com
collection
433
ITEMS
5.8M
VIEWS
by wix.com
collection
eye 5.8M
chicagotribune.com
chicagotribune.com
collection
430
ITEMS
16.1M
VIEWS
by http://chicagotribune.com/
collection
eye 16.1M